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Abstract 

Typical properties of computing circuits composed of noisy logical gates are studied using the 
statistical physics methodology. A growth model that gives rise to typical random Boolean func- 
tions is mapped onto a layered Ising spin system, which facilitates the study of their ability to 
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represent arbitrary formulae with a given level of error, the tolerable level of gate-noise, and its 
dependence on the formulae depth and complexity, the gates used and properties of the function 
inputs. Bounds on their performance, derived in the information theory literature via specific 
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phase transitions. The framework is employed for deriving results on error-rates, function-depth 

and sensitivity, and their dependence on the gate-type and noise model used that are difficult to 



obtain via the traditional methods used in this field. 
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I. INTRODUCTION 



Computation as a physical phenomena takes many forms including classical logical cir- 
cuits, quantum computing and biological neural networks. Noise is present in all practical 
computing systems and is a source of error with an immediate effect on the ability to repre- 
sent specific functions and operations and the viability of some computing paradigms. The 
main sources of errors in classical computing circuits based on semiconductor technology 
are heat generation, cosmic rays and production defects The impact of noise becomes 
even more dramatic as the drive towards miniaturization of computer components causes the 
circuits to become more complex and of large scale The presence of decoherence- noise 
in quantum computers is also a significant obstacle for exploiting their full computational 
power The effects of noise on computing and information processing in other systems, 
such as biological neural networks, which are inherently noisy, remain poorly understood. 

One of the first to study the effect of noise in computing systems was von Neumann [2I 
who attempted to explain the robustness of biological computing circuits by representing 
them as logical circuits comprising conventional Boolean logical gates. His model represented 
neural activities by a circuit (or formula) composed of e-noisy Boolean gates; he suggested 
alternative gate-constructions to limit the resulting noise and analyzed the maximal noise 
tolerated before the generated functions become random. 

Before progressing any further, a few formal definitions are required: (i) A circuit may 
be regarded as a directed acyclic graph in which the nodes of in-degree zero are either 
Boolean constants or references to arguments, the nodes of in-degree k > 1 are logical 
gates of k arguments and the nodes of out-degree zero correspond to the circuit outputs, 
(ii) A formula is a single-output circuit where the output of each gate is used as an input 
to at most one gate, (iii) The e-noisy gate is designed to compute a Boolean function 
a : {— 1, l} fc — > { — 1,1}, but for each input S G { — 1, l} fc there is an error probability 
e such that a(S) — > —a(S). To simplify the analysis, error-probability is taken to be 
independent for each gate in the circuit. Clearly, a noisy circuit (e > 0) cannot perform any 
given computation in a deterministic manner: for any circuit-input there is a non- vanishing 
probability that the circuit will produces the wrong output, (iv) The maximum of this error 
probability 5 over all circuit-inputs determines reliability of the circuit. 

In his paper, von Neumann showed that reliable computation (5 < 1/2) is possible for 
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a sufficiently small e |3| and demonstrated how reliability of a Boolean noisy circuit can 
be improved by using gate-constructions based only on e-noisy gates. There had been 
little development in the analysis of noisy computing systems until the seminal work of 
Pippenger |4| who addressed the problem from an information theory point of view. He 
showed that if a noisy fc-ary formula is used to compute a Boolean function / with the error 
probability 5 < 1/2, then (i) there is an upper bound for the gate-error e(k) which is strictly 
less than 1/2 and (ii) there is a lower bound for the formula-depth d(k,e,S) > d, where d 
is the depth of a noiseless formula computing /; the depth of a formula being the number 
of gates on the longest path from an input node to the output node. In comparison to its 
noiseless counterpart, a noisy formula that computes reliably has greater depth due to the 
presence of restitution-gates, implying longer computation times [4]. 

A number of papers have followed and extended Pippenger's results. For instance, similar 
results were derived for circuits by Feder {5], who also improved the bounds obtained by 
Pippenger for formulae. The exact noise thresholds for fc-ary Boolean formulae were later 
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determined in |6j, [7[ (for odd k only). The source of the parity restriction on k originates 
from use of a specific gate in the corresponding proofs, the majority gate (MAJ-&), for 
constructing noisy formulae; these gates have shown to be optima, for preserving a si„g.e 
input-bit of information [7\. For formulae constructed from gates with an even number of 
inputs only the noise threshold for 2-input NAND gate formulae was computed exactly jg]. 
A recent result [9] suggests that this threshold is an exact noise threshold of general 2-input 
gate formulae. 

Against this plethora of results from the information theory and theoretical computer 
science literature, our aim is to provide an alternative view based on a statistical physics 
framework, which we believe offers a powerful methodology that can recover and extend 
existing results to provide insight beyond what is accessible via the information theory 
methodology. The latter mainly rely on specific circuit constructions and methods that 
correspond to the worst case bounds. In contrast, our emphasis is on the typical case 
analysis of noisy circuits, which facilitates the study of properties at any depth and offers 
flexibility in extending the results to any distribution of logical gates. 

The analysis in the current paper is based on path integral methodology, specifically 
tailored for this task and originated in the statistical physics of disordered systems. It 
complements other methods that have been successfully employed in the study of similar 
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problems from theoretical computer science and information theory 10] ranging from classi- 
cal combinatorial optimization problems (graph coloring, k-SAT, reconstruction on trees and 
graph-isomorphism to name but a few) to source and channel coding ll|, but is arguably 
more appropriate here due to the directed nature of the formulae studied. As in the previous 
cases, we believe that our understanding will be significantly enhanced by interaction across 
disciplines [lo| . 

The study of noisy computing requires the generation of typical functions. Apparently, 
constructing typical functions by randomly connecting Boolean gates is not trivial and con- 
stitutes an area of research on its own right. Most of the familiar paradigms in the theoret- 
ical computer science literature identify gates or processes that can represent any arbitrary 
function, but when applied at random they tend generate trivial functions showing weak 
dependence on the input variables. To generate typical formulae, which compute all Boolean 
functions with uniform probability, using randomly generated circuits, we employed a variant 
of the growth process suggested by Savicky [12| that, under very broad conditions, produces 
typical functions as the depth of the formulae becomes large. 

The remainder of this paper is organized as follows. In section [III we discuss generation of 
typical Boolean functions and the model we employ for generating them. In section III II we 
define our model of noisy computation used for the analysis followed by the derivation of the 
corresponding mean-field theory in section IIV1 Results obtained by applying our method to 
random formulae which use single gates or distribution over gates are presented in section 
IVl followed by a summary and discussion of future work in section [VH Technical aspects of 
the calculations which lead to our theoretic results are provided in the Appendixes lAllCl 



II. RANDOM BOOLEAN FUNCTIONS 



To investigate the effect of gate-noise on circuits representing random Boolean functions 
one should first identify a method for generating such circuits using basic logical gates. 
The importance of random Boolean functions is in the fact that they facilitate the study of 
average case properties, in contrast to the traditionally-studied worst-case scenario jisl ]. 

A common approach to represent a random Boolean function is by constructing a random 
Boolean circuit or formula. However, finding a circuit representation of a Boolean function 
using a particular set of gates and of a bounded size is considered a difficult problem. The 
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majority of methods designated for this task use covering or bi-decomposition as their basic 
procedure [ijj]. Applying the covering method results in a disjunctive normal form (DNF) 
representation. The DNF or its dual CNF (conjunctive normal form) is a depth-2 formula 
with AND and OR gates used as internal nodes and with the input Boolean variables and 
their negations distributed on the leaves. However, random DNF (CNF) formulae offer very 
low sensitivity to the input values and any attempt to construct them at random is 
likely to produce a highly uncharacteristic random Boolean functions. 

Another approach is based on a sequential bi-decomposition of the random function to be 
implemented. In this approach, one finds a circuit representation of the Boolean function by 
reducing its dependence on a single variable at each branch of a tree, sequentially. At each 
step, the Boolean function is decomposed into two simpler functions, of the remaining vari- 
ables, that consider the two possible values of the given variable; this procedure is repeated 
until a circuit representation is found. The resulting representation may be suboptimal and 
it is not clear how to randomize this procedure in order to produce typical Boolean functions 
for a given set of simple gates. 

The most studied methods of generating random Boolean functions use random tree 
generation or a growth process as their core procedure. We will briefly introduce two of these 
methods. In the first method, a rooted k-ary tree is sampled from the uniform distribution 
of all rooted fc-ary trees; the leaves of this random tree are then labeled by the reference 
to the Boolean variables and the internal nodes are labeled by the Boolean gates used. 
Lefmann and Savicky used this construction to investigate typical properties of large random 
Boolean AND/OR formulae 16| and obtained bounds on the probability P(f) for a random 
formula to compute a given Boolean function /. These bounds were improved in follow-up 
studies [17I. fis|. which also showed that for a small number of inputs n the AND/OR model 



results in very simple functions [17| with high probability. They also suggested that this 
behavior becomes even more pronounced for large n. 

The second method uses the following growth process: Firstly, one defines an initial 
distribution over a set of simple Boolean functions. Secondly, and in further steps, the 
formulae chosen from the distributions defined in previous steps are combined by Boolean 
gates. One such process, described by Savicky [12j , uses only a single Boolean gate a and is 
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defined by the recursion on the set of formulae Af. 



A — {1, — 1, Si, . . . , S n , —Si, . . . , — S n , } 

A t+ i = {a((j>i, . . . , fc ); fa e A t for j = 1,2,..., k}. (1) 

Savicky showed, under a very broad conditions on a, that the probability of computing a 
Boolean function by a formula <p £ ^-i tends to the uniform distribution over all Boolean 



functions of n variables when £ — > oo 



121 ] . Furthermore, depending on the initial conditions 



A and the gate a the process converges to a single Boolean function or to the uniform 



distribution over some class of Boolean functions 

In this framework, all Boolean functions of n variables are represented with equal statis- 
tical weight when £ — > oo, but the number of gates in formulae grows exponentially with the 
formula depth £. Here, in order to tame this explosion in the number of gates, we propose a 
layered variant of the Savicky growth process. The first step in our process is to sample ran- 
domly and uniformly exactly N entries of an input vector S = (S®, . . . , S° N ). In the second, 
and all subsequent steps for £ = 1, . . . , L — 1, we construct a vector S + = (S[ +1 , . . . , S^ 1 ) 
where the i-th. entry Sf +1 is an output of the gate (^(S^, . . . , S { ) with k input-indices sam- 
pled uniformly from the set of all possible (unordered) indices {h, ■ ■ ■ ,ik}- The result of the 
process is the layered N x (L + 1) Boolean circuit shown in Figure [TJ (left construction, in 
blue). For large N, the variable S[ in our model corresponds to the output of a random fc-ary 
of depth t, which computes a Boolean function {— 1, 1}^ — > {— 1, 1}. We expect that in the 
limit iV — > oo, with £ 6 0(N°), the statistical properties of the formulae generated in our 
process and in the Savicky's growth process are equivalent; this is supported by the results 
reported later. The advantage of using the layered representation is that it allows us to 
explore the typical behavior of noisy random Boolean formulae using methods of statistical 
physics. 

While the vector S represents randomly sampled single entries, one would also like to 
study cases where entries are statistically dependent and are sampled from a smaller set. To 
cater for a possible higher level of correlation, the 0-layer boundary conditions are generated 
by selecting randomly S entries from members of the finite set S 1 = {S[, . . . , S^}. This 
allows to investigate the properties of the functions generated and their dependence on 
properties of the set S 1 . 
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FIG. 1: (Color online) The model of two coupled systems with identical topology and different 
inverse temperatures /3 and (3 — >oo. Gates are indicated by squares, S and input nodes by circles. 
Blue indicates noiseless gates, red noisy gates. 

III. MODEL 

As described in section [Til the noisy computation model consider here is a feed-forward 
layered N x (L+l) Boolean circuit. The layers in the circuit are numbered from (input) to 
L (output). Each layer £ G {1, . . . , L} in the circuit is composed of exactly iV e-noisy, fc-ary 
Boolean gates. Noise at gate a\ on site (i,£) operates independently and in a stochastic 
manner according to the microscopic law (see Appendix fA|) 



where (3 = 1/T is the inverse temperature, related to the gate noise e via tanh (3 = 1— 2e. The 



when (3 — > oo (e = 0). Our model is acyclic by definition, so given the state of gates at layer 
£, gates at layer £ + 1 operate independently of each other. The latter suggests that the 
probability of the microscopic state S°, . . . , S L , where S G { — 1, 1}^, is just a product of 
equation (j2J) over all sites and layers in the circuit. Furthermore, to investigate the properties 
of noisy formulas we consider two copies of the same topology, shown in Figure dj but with 
different temperatures (3 < oo (noisy) and (3 — > oo (noiseless), comparing the two will enable 
us to study the effect of noise on the resulting functions. Following similar arguments to 
those of the single circuit case, the probability of microscopic states in the two systems are 
given by 




e /3Sfaf(^ r \...,S^) 




gate-output is completely random when /3 — > (e = 1/2) and completely deterministic 
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(3) 
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where 



A? 



ns\s- ) = llj^^^^. ^., r (4) 

The set of connectivity tensors {^f where , G {0,1}, denotes connections in 
the circuit. The conditional probability P(S' IS 1 ) is the same as in equation (J3J) but with 

The sources of disorder in our model are the random connections, random boundary 
conditions and random gates. The former two arise in the layered growth process described 
in the last two paragraphs of section [TT1 The basic step in this growth process is the 
addition of a new gate with probability P(A^ • ) = -^5 A i,i , + (1 — jL)5 A e,i n of 

ji,—,jk iv A jl dk ,i JV A j 1 ,...,j k ' K3 

being connected to exactly k gate-outputs on the previous layer £ — 1. This procedure is 
carried out independently for all gates in the circuit giving rise to the probability distribution 



L,N 



= ^7 n 



1,1=1 



A 



ii.---.ife 



N 1 , 1 



n 



(5) 



where is a normalization constant. The Kronecker delta function inside the definition 

fl5]) enforces the constraint j k A h,-,jk = ^' *' e ' ^ e S a ^ e on s ^ e (^•O i s mapped to 

exactly one element from the set of all possible output-indices {ii, . . . , ik] from the previous 
layer. Other sparse connectivity profiles can be easily accommodated into our framework 
by incorporating additional constraints into the definition ([5]) via the appropriate delta 
functions. 

Random boundary conditions in the layered growth process are generated by selecting 
members of the input set S 1 , where |S J | G 0(N°), with probability r^n, and assigning them 
to the input layer 0. The boundary condition is identical for two systems which leads to the 
distribution 

N 

P(S°,S°\S I ) = l[5 s o. iSL 5 § o ]SO (6) 

i=l 

where {n^} are independent random indices pointing to the members of input set S 1 with 
probability P{rii) = j^j. Further correlations can be introduced by defining the probability 
function P^S 1 ). 

In addition to the topological disorder, induced by the growth process, we assume that 
the gate a\ added at each step of the process can be sampled randomly and independently 
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from the set G of fc-ary Boolean gates. Under this assumption the distribution over gates 
takes the form 



L,N 



p({<4}) = n p («') ( ? ) 

e,i=i 

where P(a|) = EaeG with EaecP" = 1 and p a > 0. 

IV. METHOD 

To compute the probability distribution fl3]) directly for a circuit of finite but significant 
size is difficult. However, the structure of equation is similar to the one that describes 
evolution of the disordered Ising spin system This similarity becomes apparent if one 
regards the layers in our model as discrete time-steps of parallel dynamics. A common way 
to deal with the probabilistic objects that take this form is to use the generating functional 
method of statistical mechanics 20} . The generating functional for the current model is 
given by 

r[V;<0] = (e^'M^m^ (8) 

where the shorthand (. . .) denotes the average over the joint probability ([3]). The generating 
functional ([8]) can be regarded as a characteristic function of ([3]) from which moments of 
the distribution can be obtained by taking partial derivatives with respect to the generating 
fields {ipf, ipf}, for example (SfSf) = — h m ^ ^,^q a %. , ^l^i w\- Following prescription 

of 



20], we assume that for iV — > oo the system is self- averaging and compute T[ip;i^], where 
[■ ■ ■ ] denotes an average over the disorder. The disorder-averaged generating function ([8]) 
gives rise to the following macroscopic observables 

™W = 7?E^> = lim 7tY,7r T ^* ( 9 ) 



N ^ N 



c ® = a7 E <^> = - iim 1 7nr T[ ^ ^ ] 

N ~t V.V'^o N ~t tyty 

where m{€) is the average activity (magnetization) on layer i and C(£) is the overlap between 
two systems. Averaging over the disorder in ([8]) leads to the saddle-point integral (see 
Appendix IB1 for details) 

r = J{dP dp dn drije^^ '^ (10) 



where \I/ is the macroscopic saddle-point surface 

L-l L-l „ 

*[...] = i^^p f (s,5)p^(5,5) + /dxdidwnV.^w)^ 

£=0 5i s <=0 ^ 

L-l fc 



(11) 



' {Mi}J' =1 



dx dx dw fi £ (x, x,w) (e 



-i{xa({S J })+xa({5 J })+w} 



+ ^ P(n) log J {dH dx dH dx} J^Yl M n\ H , *\ H , x\ u; S, S] , 

S,S 

where (-) a represents an average over gate distribution and M is an effective single-site 
measure 

L — 1 „ § „ - B 



Mr. 



£=0 



\x e H +ix e H +ps e + 1 H +ps l + 1 H 



(12) 



xe 



log 2 cosh (pH^ -log 2 cosh (pH ^ -iP\s e ,S e ) -i£l e (x l ,X l ,uj e+1 ) 



The generating fields if), if) have been removed from the above as they are no longer needed. 
For iV — > oo the path- integral (II 01) is dominated by the extremum of the functional ^> [. . .} of 
equation Functional variation of ffTTT) with respect to the order parameters {P, P, Q, &} 
gives rise to four saddle-point equations 



J S e :S 



k 



p\s, s)=ij2 e 6 ^ 6 s,s n [ p &> & 

i=1 {Sj,Sj} i^i 



(13) 
(14) 



x / dx dx do; Q (x, x,oj) (e 



{ m ({S j })+m({S j })+ W }\ 
/ a 



U'- u/ +1 )) 



n e (x, x, u) = E p ( n ) - - 

-i{xa({S , J })+£a({5 J })+w} 



{s i ,s i }i=i 



(15) 
(16) 



where (•••)a/„ is the average over the probability distribution resulting from (jl"2l . The 
saddle-point equations f|T3l - f|T6l) can be simplified significantly (see Appendix O for details) 
and it turns out that in order to solve this problem we only need to compute the order 
parameter f[T3"j) . The physical meaning of this order parameter is given by P (S,S) = 



limN^oo J2iLi s e ;S$s e - s) Is 1 ? i- e - the disorder-averaged joint probability of sites in the 
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two systems. The single-site effective measure ( 1121) also benefits from the simplification; in 
particular, if we integrate out the continuous variables in (JT2"|) we are led to the expression 

L-l ( k 

M n [S L , S L ,..., S°, S°] = S S o. sl 8 §0;SO n En [^(^ 4)] 

<=0 I {5,,5 J }i= 1 
e ^a({S,}) e $S e+1 a({Sj}) 




2cosh/3[«({S , ,})]2cosh/3[«({S',})] 

Using equation ffTTl) the macroscopic observables ([9]) can be easily computed from the joint 
probability distribution ( TT3|) . resulting in the set of equations 



^+ i )=En 

{^■}i=i 

c(€+i)= Yi n 

{5 J ,5,}i=l 



I{1+^)} 



(tanh[/3«(^,...,^)]) Q (18) 

(19) 



x ^tanh[/3a(5"i, . . . , S k )] tanh[/3a(Si, . . . , £> fc )] 

where the magnetization rh(£) is computed by a similar equation to flTSJ) but with /3 — >- /3. 
The initial conditions for the above system of equations are given by m(0) = m(0) = 

The connectivity profile considered in our model leads to a simple mean-field theory, where 
the macroscopic behaviors of the two copies of the same system is completely determined 
by the set of observables {m(£),m(£),C(£)}] which relate to the order parameter (fT51) via 
P e (S, S) = |(1 + Sm{£) + Srh(£) + SSC(£)), while the single system behavior is described 
by {m(£)}. Furthermore, since (Y[j Sf.) — >Yij(^i ) f° r finite j, when N oo (this can be 
shown by the direct computation via (jBJ)) the spins on layer I are uncorrelated. The reason 
for this behavior is that in our model, the site (i, £) is a root of a full fc-ary tree growing from 
the input layer I = 0, which in turn points to the input set S 1 . The loops in the circuit are 
rare, so that trees can be regarded as random Boolean formulas and when presented with 
the input, operate independently of each other. The output of a typical formula at layer i 
is determined by the probability P e (S). 

The overlap order parameter of equation ([9]) is related to the normalized Hamming dis- 
tance D{£) between the states S E and S via the identity D(£) = |(1 — C{£)). This al- 
lows one to define the order parameter A(£) = linig |(1 — C(£)), used to probe sen- 
sitivity of the circuit with respect to its input, an indication to the complexity of the 

11 



0.5 
0.42 

e* 

0.35 
0.27 

0.17 

3 17 37 59 83 101 

k 

FIG. 2: (Color online) Phase diagram of a circuit based on MAJ-k gates. 

functions represented by the given circuit. The Hamming distance D(£) is also related 
to the probability P{S\ ^ and facilitates the estimate of the error probability 5{£) 
on the £-th layer of a noisy circuit. More specifically, we define this error probability 
S(£) = max s ; linig^^ |(1 — C(£)), comparing the maximal error between the noisy and 
noiseless version of the same circuit with respect to all possible inputs. Obviously, in the 
absence of noise (/3 — > oo) one trivially obtains 5(£) = for all £. 

V. RESULTS 

In this section we apply equations (I18|19p to the formulae constructed from a particular 
single gate a and for a given distribution over gates P(a). 




A. MAJ-fc gate 

1. Critical behavior 

In this section our choice of a is a universal majority gate with k inputs (MAJ-A;). The 



reasons for choosing this gate are twofold. Firstly, it was proved in jfj], |7[ that the majority 
gate is optimal for the noisy computation in formulae. Secondly, formulae constructed 
from the majority gates can in principle compute any Boolean function [12|. A convenient 
representation of the MAJ-k gate is given by the identity MAJ(Si, . . . , Sk) = sgn[J^ =1 Sj], 
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where k is odd. Inserting this identity into the equations (lisp and (I19p and computing the 
spin averages leads to 



n=0 ^ ' 



1 + m(£) 
2 



m(£) 



2 



k—n 



sgn [2n - k] (20) 



C(/ + l) = (l-2c) £ J P fcl (~l,-1) P fc2 (l,~l) (21) 

fei+..+fc4=fc 



X 



P fc3 (-1, 1) P fc4 (l, 1) sgn [k 1 -k 2 + k 3 -k 4 ] sgn [kx + k 2 -k 3 -k 4 



where P(S, S) = |(1 + Sm(f) + 5m(£) + S§C(£)). To obtain equations fl201T2Tj) we have 
used the identity tanh (3 = 1 — 2e which relates the gate-error e to the inverse temperature 
P. Also, we have taken the limit (3 — > oo (e = 0) to compare noisy circuit outputs to their 
noiseless counterparts later on. 

For now we concentrate only on equation (|20|) which describes the evolution of magne- 
tization from layer to layer. The point m(oo) = is always a stationary solution of this 
equation, i.e. m(£ + 1) = m(£) = m(oo). Expanding equation ( 12 Op around this stationary 
solution gives the critical noise value e*(k) = 1/2 — 2 fc ~ 2 /A;( ( . fc fc ^ 2 ), identical to the results 



of 



7|; at the critical noise the asymptotic solution m(oo) = becomes unstable and two 



stable solutions ±771(00) (e.g., for k = 3 we find m(oo) = ±y j5§i |2l|) emerge. In the case 
when e > e*(k) the magnetization m(£) decays to when £ — > 00. For the e < e*(k) we have 
lim^^oo m(£) = ±m(oo) where the positive and negative stationary solutions correspond to 
the positive and negative initial magnetizations m(0) = t^t J^ses 1 ^1 respectively. Thus the 
critical noise level e*(k) separates the unordered phase of the system from ordered one (see 
Figure E]). 

The relation between the new stable solutions and the reliability of the computation 
follows from the ability to preserve one bit of information presented at the input, by setting 
S 1 = {S}; the phase transition observed in equation fT20|) implies that the circuit can preserve 
one bit of information for arbitrarily many layers only when e < e*(k). The probability of 
an error P l (—S) = |(1 — Sm(£)) is a measure of how well this one bit is preserved after 
passing through £ layers. A complicated computational task may require significant number 
of layers, hence only relatively simple operations can be performed by the circuit reliably 
when e > e*(k). 

Now we turn to equation (1211) which describes evolution of the overlap between two 
systems. The initial conditions are the same for both systems, so we have m(0) = m(0) and 
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C(0) = 1. The magnetization in the noisy system (e < t*{k)) converges to ±m(oo) and for 
the noiseless system we have m(oo) = ±1, depending on the sign of m(0). Inserting these 
stationary points into the equation (T2T]) results in C(oo) = ±m(oo). The overlap C(oo) 
relates to the probability of error 6(00) = \{\ — C(oo)). Thus the error 8 (06) is bounded 
below 1/2 only when e < e*(k). 



2. Boolean functions generated 

The analysis of equation fl20l) can also reveal the type of Boolean functions generated 
in the layered growth process. In particular, in the noiseless case (e = 0) the stationary 
solutions of this equation are given by 771(00) = 1 and ra(oo) = — 1 which correspond to the 
initial conditions m(0) > and m(0) < respectively. For m(0) = we obtain 771(00) = 0. 
Each site in our model can be associated with an output of the formula that computes 
some Boolean function. The average formula on layer i provides outputs S with probability 
P e (S) = |(1 — Sm(£)). This suggests that for the noiseless case e = the average formula 
on layer I converges to a random Boolean function 

+ 1 ifm(0)>0 

±1 withprob. V 2 if m(0) = (22) 
-1 ifm(0)<0 

where m(0) = r^n ^2 SeS i S. This means that depending on the initial conditions the for- 
mulae converge to a single Boolean function or to the uniform distribution over some set 
of functions. For example, if we take S 1 = {—1, —S[, —S^} then the formulae converge to 
the NAND function. Taking S 1 = {—S[, — Sf,}, on the other hand, gives us uniform dis- 
tribution over the inverse functions —S[ and —St, as follows from the majority property of 
the gate. In general, when m(0) = 0, is difficult to say if the formulae compute all Boolean 
functions in the set (122 j) or only the subset of these functions. However, this result 



is consistent with the study of Savicky 12j where the majority gate forms the basis of the 
growth process that generates random Boolean formulae. In particular, it has been shown 
that when S 1 = { — 1, 1, S(, . . . , S^, —S(, . . . , — S^} the formulae in the stationary state of 
the process compute all Boolean functions of n variables. Furthermore, the equation f )22|) 
is also consistent with results reported elsewhere [3] where the same growth process is 
considered for various initial conditions S 1 . In particular, for S 1 = {— 1, 1, S( , . . . , S^} the 
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formulae converge to the MAJ-n function when n is odd and to the uniform distribution 



over so-called slice functions when n is even 



131 ] . The same happens when the constants 



{ — 1, 1} are removed from the set S 1 13]. The result of equation fl22|) can be written in a 
more compact form F = sgn[^" =1 9iS\ + 9o], where 9i G Z, using the definition: sgn[0] = ±1 
with probability 1/2. If £™ =1 e t S{ + 9 ^ for V (S(, ...,S£)e {-1, l} n then all formulae 
in the circuit converge to a sing le linear threshold function which can compute any linearly 
separable Boolean function 22]. 



3. Sensitivity of the generated functions 

We will now turn to equation (12 ip when m(0) = m(0) =0 and C(0) = 1. For the noiseless 
case e = the stationary solution of this equation is given by C(oo) = 1. This solution, 
however, is unstable and a small perturbation to the initial state C(0) = 1 leads to the 
stationary state C(oo) = 0, which is stable. This implies that the circuit is very sensitive to 
its input when m(0) = 0. In particular, the Hamming distance A(£) = |(1 — C(£)) increases 
for small perturbations to A(0) (see Figures [3] (a)), i.e. a small perturbation to the input 
is amplified by the circuit. This in turn means that when e > the circuit also amplifies 
the noise-perturbation and the error 8(£) is growing. The error, however, can be kept under 
control for many layers by making e sufficiently small (see Figure [3] (b)). 



4- Dynamics 

We will now examine how the computation in the circuit proceeds from layer to layer. 
As an example we take S 1 = {Si, . . . ,Sn}, i.e. for e = the circuit computes the MAJ- 
11 function, and study the evolution of magnetization m(£) and error 5(£). The initial 
magnetization |m(0)| takes its values from the set {1,9/11,7/11,5/11,3/11,1/11} for this 
choice of S 1 . The input with the smallest magnetization in this set is very important. On 
the one hand, when e = the initial state with the smallest |m(0)| is also the furthest from 
the stationary state |m(oo)| = 1. So it will take for the magnetization m(£) the largest 
number of layers to converge for this input. On the other hand, when e > the input with 
the smallest |m(0)| is more likely to be destroyed by the noise. For these reasons, in what 
follows we study the evolution of the magnetization and errors only for m(0) = 1/11. 
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FIG. 3: (Color online) (a) Evolution of the Hamming distance as a function of the layer i for k = 3 
(solid line) with input mismatch A(0) = 10~ 3 , 10 -4 , 10 -6 (left to right) and for k = 5 (+) and 
k = 7 (x) with A(0) = 10~ 6 . (b) Phase boundaries for gate-noise e at layer L when k = 3,5,7, 
using the same symbols. 

In Figure H] (a) we examine how the magnetization m(£) evolves from layer to layer in cir- 
cuits with MAJ-3, MAJ-5 and MAJ-7 gates. We observe that when e = the magnetization 
converges to its stationary value m(oo) = 1 relatively quickly. Since we use m(0) = 1/11 
the convergence to m{£) — 1 indicates that all formulae in the circuit compute the MA J- 11 
function. For noise values e > the speed of convergence is decreasing as e increases and 
becomes very slow as we approach e*(k). In general, increasing k (0 < e < e*(k)) leads to a 
reduction in relaxation times because of the inequality F^ +2 [m] > F^[m] > m, where F^[m] 
is the right hand side of equation (|20|) . Finally, when we increase the noise level to e ^> t*{k) 
the magnetization relaxes to its stationary value exponentially fast. 

Figure |4](b) shows the evolution of the error S(£). In the region < e < e*(k) we observe 
two distinct stages in the dynamics. Initially, the error is increasing until it reaches its 
maximum value. Note that this happens before the MAJ-11 function is computed exactly 
when e = (see Figure H] (a)). Also, the location of this maximum is only weakly affected 
by noise. These two observations suggest that initially the inputs to the gates are very inho- 
mogeneous which leads to the amplification of noise. After the error reaches its maximum 
value the inputs become more and more homogeneous leading to the suppression of noise 
and as a result the error decreases until it eventually becomes stationary. As we approach 
the critical boundary e*(k) the number of layers needed for the error to equilibrate increases. 
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FIG. 4: (Color online) Evolution of magnetization (a) and error (b) in MAJ-/c formulae for k = 
3,5,7 (e*(3) « 0.167, e* (5) « 0.233, e*(7) « 0.271) and different noise (e) values. 

The dynamic behavior of the error changes from the non-monotonically increasing to the 
monotonically increasing when we approach the critical boundary e*(k) from below. The 
evolution of error becomes strictly monotonic when e ^> e*(k) and in this region the error 
relaxes to its stationary value 1/2 exponentially fast. 

This example is highly representative of the situation when all formulae in the circuit 
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converge to a single Boolean function. Here we can tell exactly how many layers it takes 
for the circuit to compute this function when e = 0. This number is given by L such that 
m(L) « m(oo) starting with the smallest initial magnetization m(0) induced by the inputs 
S 1 . Obviously, adding more layers to the circuit with L layers will not affect the compu- 
tation when e = 0. However, in the region < e < e*(k) we can reduce the error S(£) by 
adding more layers. This is not true for all e and m(0). The error can be reduced only 
when e < e°(k), where e°(k) is the solution of m(0) = F^ o ^[m(0)] for a given initial magne- 
tization m(0). At e = e°(k) the dynamic behavior of m(£) changes from the monotonically 
increasing (when e < e°(k)) to monotonically decreasing (when e > e°(k)). Respectively, 
the dynamic behavior of the error 5(£) changes from the non- monotonically increasing to 
the monotonically increasing. Only in the former regime one can reduce the error S(£) by 
adding more layers to the circuit. However, this strategy fails for inputs with m(0) = and 
the circuit computes more than one Boolean function. For m(0) = the probability of error 
S(£) increases towards its stationary value 5(oo) = 1/2 (m(£) = 0) even when e < t*{k). The 
error 5(£) can be bounded away from 1/2 only by reducing the gate-error e, which depends 
on the formula depth L (Figure |3] (b)). 



5. Convergence rates 

In this section we study convergence rates at e = and e = e*(k) ± Ae regions of the 
phase diagram plotted in Figure |2j The former allows us to estimate the number of layers in 
a formula, which is directly related to its size, when all formulae in the circuit converge to 
a single Boolean function. The latter probes the regime where the computation is expected 
to be very slow, but the error can be still reduced by adding more layers to the circuit. 

Firstly we study the rate of convergence when e = and m(0) = 1/n, where n e N is 
odd, and the MAJ-/c based circuit computes MAJ-n function. In general, we find that the 
number of layers needed for the magnetization to converge scales as 0(f(k) log(n)). This 
rate of convergence is consistent with rigorous results [13| for the growth process defined by 



Savicky [l2j. However, the worst case bound f(k) derived in 13] grows as k2 k with the gate 
in-degree k, while in our study we find that f(k) is decreasing with increasing k (see Figure [5] 
). Furthermore, this result holds not only for MAJ-n, but for any linear threshold function 
(with integer weights) computed by the MAJ-fc circuit. It is natural to expect that when 
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FIG. 5: (Color online) Number of layers in the noiseless MAJ-A; based circuit computing MAJ-n 
function. Theoretically obtained data-points are represented by the symbols + (k = 3) , x (A; = 5) 
and * (k = 7). Slopes of the respective straight lines (not shown) fitted to the data are 2.473, 1.595 
and 1.282, respectively. The straight line corresponds to the bound A;2 fc log(n), derived in |l^ |. 
plotted here for k = 3 only. 

k — » oo the function f(k) is vanishing and the numbers of layers L in the circuit approaches 
1. The discrepancy in the asymptotic behavior of the worst case bounds and the typical 
asymptotic behavior observed in our work is due to the average topology considered here, 
which turns out to be more realistic. 

Secondly, we study convergence rate for e = e*(k) ± Ae, where < Ae < 1. Very close to 
the phase boundary e*(k) the differences m[l + 1) — m(£) are very small and the difference 
equation f l20|) can be well approximated by a differential equation. For k = 3 the differential 

where I is continuous, which 

-l 



i 



l-2e 



c -(l-6e)l I 1-26 
1— 6e 



equation reduces to ^m[€) = — m(£) + |(l — 2e)[3m(£) — ttt 
can be solved exactly. The solution is given by m 2 (£) — , ; M 
This approach is only accurate in the region e = 1 /6 ± Ae, where it gives us the asymptotic 
form \m(£)— m(oo)| w e -7(3)Ae£_ coefficient equals 3 in the paramagnetic region and 

6 in the ferromagnetic region. Thus the convergence to the asymptotic solution m(oo) = is 
slower than to the stationary solutions m(oo) 



± 



l-6e 
l-2e 



for k = 3. The differential version 



of the difference equation f r20|) is difficult to solve analytically when k > 3 and for these 
values of k we will use a different method to estimate the convergence rate. This method 
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relies on the fact that 



const x 



dm 



< \m(£) — m(oo)|, 



(23) 



m=m(oo) 

where F^[m] is the right hand side of equation f )20|) . i.e. the distance \m(£) — m(oo)| for an 
arbitrary point m(£) is always greater than the distance for m(£) = m(oo) + Am(£). The 
lower bound (123]) can be made into an upper bound by choosing an appropriate constant 23]. 
Computing the left hand side in ( 12 3 p for the MAJ-A; circuit with e = e*(k) ± Ae leads to the 
result 



2Ae 



1 - 2e*(k)_ 



(1 - m 2 (oo)) (fc - 1)/2 } < \m{£) - m(oo)|. 



(24) 



Since 



1 - 



2Ae 



l-2e*(fc) 



(l-m^oo))^ 1 )/ 2 < 



1 



2Ae 



l-2e*(fc) 



, the convergence rate in the param- 



agnetic region (e = e*(k) + Ae) is slower than in the ferromagnetic one (e = e*(k) — Ae). 
The latter is due to the amplification of thermal fluctuations which are only suppressed for 



£ — > oo. For large k the critical noise e*(k) can be approximated by e*(fc) r 
Inserting this into the equation (124"|) for m(oo) = gives the asymptotic form 

\m(£) - m(oo) \ ^ e- o{k0) ^ Ae \ 



(25) 



from which is clear that increasing k speeds up the convergence in both paramagnetic and 
ferromagnetic regimes. 

6. Hard noise 



The model studied so far can be regarded as a model of computation where errors result 
from single-event upsets (soft noise). In real integrated circuits [l], the imperfections intro- 
duced into the circuit during the production process are an additional source of permanent 
errors (hard noise). 

A natural way to introduce hard noise into our model is to define quenched random 
variables where P(£f) = £><% ; -i + (1 — £>)<% ; i, which can invert the gate output a\ 

permanently. Using transformation a\ — > £faf in equation (T5]) and following the steps of 
calculation in section IIVI we find that the inclusion of hard noise in our model leads to 
(1 — 2e) — > (1 — 2p)(l — 2e) in equations (TSU]) and (|2"T|) . As a result, the effect of quenched 
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FIG. 6: (Color online) The critical noise level e* as a function of k for the perceptron (dotted line) 
and MAJ-fc (solid line) circuits. 

noise is to reduce the critical noise e*(k). In particular, the new critical noise value is given 

by e*(k,p) — \ — 2 V 2 fc _i n when < p < e*(k, 0) and e*(k,p) = when p > e*(k, 0). 

The hard noise can also be introduced by making a fraction of gates insensitive to the 
inputs, i.e. gates that produce constants. In particular, by taking P(a) = po^a;MAj-k + 
P-S a -i + P+8 a -i, where p± are the probabilities of constant ±1 outputs and p ,p_,p + > 

with Po + P- + P+ = 1, equation ffTSl) for p + — p_ = results in e*(k,p ) — | iT k-i \ 

when < (1 -p )/2 < e*(k,p = 0) and e*(k,p ) = when (1 -p )/2 > e*(k,p = 0). So 
the introduction of constant gates reduces the critical noise value e*(k) when p + = p_ by 
effectively reducing the number of active gates. For p+ — P- ^ the effect of hard noise 
is more drastic. For e = the circuit, irrespective of its input, is biased towards one of its 
outputs ±1, depending on the value of p + — P-- 



7. Threshold noise 



The MAJ-fc function can be seen as a special case of a linear threshold function. The 
linear threshold functions (or binary perceptrons) are widely used in the modeling of the 
neuronal activities of the brain such as memory and learning 24]. The noise in these models 
is usually introduced via random thresholds. The stochastic process for the simplest model 
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of this class is governed by the algorithm 24] 



Si = sgn 



.i=i 



(26) 



where rjf G R are independent random variables drawn from the distribution P(rj) — |[1 — 
tanh 2 (??)], which generates the state update of individual neurons (on site i) at step i. The 
stochastic algorithm (126]) gives rise to the microscopic law 



- r l°il°il ) • • • 1 °; 



2cosh[/3ELC 1 ] ' 



(27) 



which in the limit of /3 — > oo corresponds to the noiseless MAJ-fc gate. Using the microscopic 
law (|27p in the model of noisy computation Q requires only minor change to the calculations 
in section IIVI and leads straightforwardly to the corresponding equations for magnetization 
and overlap 



m(l + l) = 



n=0 



n 



E 



1 + m(£) 



k\ 

k\\ x .. x A; 4 ! 



m{£) 



k—n 



tanh[/3(2n - k)] 



P kl (-1,-1) P k2 (l,-1) 



(28) 
(29) 



fci + ..+/C4=fc 

x P fca (-l,l) P fc4 (l,l)tanh[/3(fc 1 -A; 2 + fc3-fc4)]sgn [h + k 2 -k 3 -k 4 



In the limit (3 — > oo(e = 0) equations (12811291) and (1201121 p . with e = 0, are identical. So that 
all the results derived for the noiseless MAJ-fc circuit are also valid here. 

The macroscopic behavior of the model with noisy thresholds, however, is different from 
the noisy MA J- A; model for any e > 0. Analysis of equation (1281) reveals that the point 
m(oo) = is always a solution of this equation. Expanding equation (1281) around this point 
leads to the condition 

A: 



n=0 



n 



\2n — k\ tanh/3|2n — k\ 



(30) 



where m(oo) = becomes unstable and two stable ±m(oo) solutions emerge. In Figure |6j 
we compare the resulting phase boundary with that of the MAJ-/c based circuits. The 
MAJ-fc gate is more resilient to the threshold noise (I26p than to the flip noise (1A1[) . This 
is not surprising since the effect of flip noise on the MAJ-fc gate (inverting the gate-output 
regardless of the input) is more drastic than the effect of threshold noise where gate-outputs 
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Sf corresponding to inputs with high input-magnetization | Y^j=i ^tj X \ are l ess likely to be 
flipped. Furthermore, as k — > oo the critical noise level e*(k) in both models approaches 
1/2 as 1/2 — e*(k) = 0(/c -7 ), but for the threshold noise model it can be shown that 7 = 1 
while for the model with flip noise 7 = 1/2. 

For the noisy threshold model considered here the evolution of magnetization and 5-error 
is qualitatively similar to the evolution of these order parameters in the flip-noise model. 
However, we find that the convergence to the stationary state is much quicker in the noisy 
threshold model. For large k it can be shown that the convergence to all stationary states 
is dominated by 

\m(£) -m(oo)\ ~ e -° (fc ° )fcArf , (31) 

which is significantly quicker than (I25p . 

Finally, we note that the magnetization equation f[2"gj) is exactly equal to the magnetiza- 
tion equation derived for the parallel dynamics of an Ising ferromagnet on fully asymmetric 



Bethe lattice 
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26[ if the layers in the circuit-model are regarded as the time-steps of 
parallel dynamics. This suggests that the site-time topology (recurrent network) generated 
by the parallel dynamics is similar to the topology of layered networks considered here when 
N — > 00. Based on this observation, which was exploited for instance in |27|. we expect 
that the computation performed by the recurrent and layered networks to be the same at 
least for m(0) 7^ 0. The only differences (if any) can arise from the non-vanishing connected 
correlations between the different times (layers) in the recurrent network. 



B. NAND gate 

Here we apply the theory to the formulae constructed by 2-input universal NAND gates 



which, according to js, 9], are optimal for the noisy computation by a 2-ary Boolean formulae. 
The NAND gate can be represented as the linear threshold function sgn[— Si — S% — 1] . Using 
this representation in the magnetization (ITS]) and overlap (119]) equations, with j3 — > 00 and 
tanh/3 = 1 — 2e, one obtains 

m(£+l) = i(l-2e)[(l-m(£)) 2 -2] (32) 

C(£+l) = -(l-2e) [l + 2m(£) + 2m(£) + 2C(£)-m 2 (£) + 2m(£)m(£) 
-2m(£)C(£) -m 2 {£) -2m(£)C(£) +C 2 (i)] (33) 
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FIG. 7: (Color online) Magnetization m and error 5 as a function of gate-noise e in NAND-gate 
formulae. The dependence of 5± on the noise level in the range 0<e< (3— \/7)/4 is shown in the 
inset. 

where m(0) = m(0) = j^j^s^s 1 ^(0) = 1. Here the equation for m(£) is identical to 
equation f l3"2"j) . but with e = 0. The magnetization equation f[3"2"j) admits only one steady 
state (m(oo) = m(£ + 1) = m(£)) solution in the region m(oo) G [—1,1]. However, the 
solution m(oo) 



1 



l-\/8e 2 -12e+5 
2e-l 



to equation (132]) becomes unstable for the values of noise 
e < e* = (3 — \/7)/4 which identifies e* as the critical threshold. Above this threshold the 
(output) magnetization on layer £ — > oo converges to the value m(oo), which is independent 
of the initial (input) magnetization m(0). Below the threshold e* the magnetization m(£) 
oscillates from layer to layer and the properties of this oscillation depends on m(0). The 
latter suggests that the circuit performs computation only in the region e < e*, in agreement 
with ja]. 

To explore the properties of this computation we follow the evolution of the probability 
P-(£) = (1 — m(£))/2 . Using this definition in the equation ( 13"2"|) we obtain the difference 
equations for P-{£) on even layers 



P_(£ + 2) = l-e-(l-2e)(l-e) 2 + 2(l-e)(l-2e) 2 P_ 2 (£) - (l-2e) 3 P^(£) 



(34) 



where -P-(O) = (1 — m(0))/2. Equation f )34p also describes the evolution of P-(£) on odd 
layers with the initial condition being P_(l) = |(1— 2e) [Pl(0) — |]. In the region P G [0, 1] 
equation (JSJ) admits three steady state solutions P_(oo) = j W8 4 £ "r 2 12e+ -, ~ 1±v/ ^~ 12e+1 -|. 
The first solution becomes unstable at the noise threshold e* = (3 — V/)/4 and the other 
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two solutions are stable for the noise values e < e*. Plotting these solutions with respect 
to the noise e in terms of the corresponding magnetization variables m(oo) gives the phase 
diagram depicted in Figure UJ The stationary solutions of equation (jMjl also allow one to 
compute the 5-error which, due to oscillatory behavior of the magnetization m(£) in (1321) . 
depends on the sign of the output. In particular the error 5(oo) takes its values from the set 



{6 _ = 6+ = 1 _ -W8^-i2e+i | when m(Q) ^ (2 _ ^ 1 _ WEH+I ) an d 

5(oo) G {1 - <?+, 1 - <L} when m(0) G (2 — v^o", 1 — ^^ ), The dependence of the 



error- functions on the gate-noise e is shown in the inset of Figure [7J 

For e = the basins of attraction of the fixed points P_(oo) G {1,0} are given by 
P_(0) G [1, ^pi) and P_(0) G (^, 0] respectively Thus for e = and £ oo the NAND 
formulae compute the Boolean function 



to the result for MAJ-/c circuit ( l2"2"j) . the variety of functions generated by the NAND for- 
mulae is rather limited. All formulae in the NAND circuit converge to the linear threshold 
function ([3"5"j) which, due to the threshold value being equal to 2 — v^5, cannot compute all 
linearly separable Boolean functions. 

As an example of a noisy computation by the NAND circuit we consider the input 
set S 1 = {Si, S2, S3}; the initial magnetization is given by m(0) = (Si + S2 + Ss)/3 G 
{ — 1,-1/3,1/3,1} and according to (j3"5]) the noiseless circuit converges to the MAJ-3 
Boolean function on even layers and to its dual on odd layers. In Figure El we plot the 
evolution of magnetization m and <5-error only for m(0) = — 1/3 where the number of layers 
£ needed for the magnetization m{£) to converge to its stationary value —1 (for £ even) is 
maximal. 

We observe that for e = the magnetization approaches its stationary value —1 in ap- 
proximately 16 layers, so all the formulae compute MAJ-3 function after 16 layers. For 
gate- noise values e > 0, the number of layers L needed for the magnetization to became 
stationary increases as we increase e towards its critical value e*, while the stationary mag- 
netization value m(oo) decreases (|m(oo)| < 1). As a result, the stationary values of the 
5-error, which is directly related to m(oo), grow monotonically towards 8(00). The error 




(35) 



where m(0) 




S, when £ is even and its inverse — F when £ is odd. In contrast 
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FIG. 8: (Color online) Evolution of magnetization and 5-error on even layers in NAND formulae 
computing MAJ-3 function. 



grows first then reduces with the addition of layers (see Figure [8]); this reduction of error is 
only possible when e < e(0) 



m 2 (0)-m(0)-3+ v /-3m 2 (0)+4m(0)+8 
2(m 2 (0)-2m(0)-l) 



-. Above the critical noise level 
e* the stationary magnetization m(oo) is independent of m(0) and the computation becomes 
unreliable. 

Following the method outlined in section IV A[ we obtain the rate of convergence to the 
stationary solutions of equation f[34"j) . In the range < e < e*; for A = e* — e we find 

3+V7 



const x exp 



-.o g ,8,^-A 



4 



+A 



< \P_(£) -P_(oo)|. (36) 



Close to the critical noise level (A — > 0) the argument of the log function approaches 
unity and the number of layers needed to converge to the stationary solution (the point 
of intersection of all magnetization curves in Figure [H]) diverges. In the opposite limit of 
A — > e* = the argument of the log function approaches zero and the convergence to 
the stationary states m(oo) = ±1 is very fast. 

Finally, we note that the results of this section give us a positive answer to some of the 
conjectures put forward in |8|], in particular: a) The threshold e* = is valid for the 
random NAND formulae with completely reliable (hard) inputs, b) The computation at 
e = e* is not possible, because equation (I3~4"|) has only one fixed point which is both unstable 
and attractive. 
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C. AND/OR gates 



In this section we study random Boolean formulae constructed from a noisy AND and 
OR gates. The noiseless version of the AND/OR model defined on unbalanced trees was 



used in past to define the probability distribution on Boolean functions 
of balanced trees was considered recently in 



16- 



The case 



In the model of computation which we consider here the gate a is sampled from the 
distribution P(a) = p^a;sgn[s 1 +5 2 +i] + (1 ~~ p)^a;sgn[Si+s 2 -i]- Using this definition in the gate- 
averages of equations ( ITS]) and ( 1T9|) gives 

m{£ + 1) = -(1 - 2e) [2p-l + 2m{£) - (2p - l)m 2 (£)] (37) 
C{£ + 1) = i(l - 2e)[l~Am(£)-Arh(£)+AC{£)+Apm(£)+Aprh(£) (38) 



-Apm[£)C{£) -Apm(£)C(£) -m 2 (£) +Am(£)m(£) +Am(£)C(£)-m 2 (£) +Am(£)C(£) + C 2 ' 

Equation (|37|) can be written in a more convenient form 

P_(£+ 1) = e + 2(1 - 2e)(l -p)P-{£) + (1 - 2e)(2p - l)P 2 {£) (39) 

where P-(£) = (1 — m{£))/2 is the probability of output taking value of —1. For e > 
equation fl39l has only one (stable) steady state solution P_(oo). Thus there is no phase 
transition in this model for any noise value e > and the information about the input cannot 
be preserved for infinitely many layers. 



The noiseless balanced AND/OR trees were studied in 28j. Here we only show how to 



recover their results 



30| from the equations (l37jl , (l3"8l and (1591. 



Firstly, we note that by setting e = in equation fl39|) one obtains the equation of 
Lemma 3.1 in 281 ]. Equation (139]) has two fixed points P_(oo) e {1, 0}(m(oo) e { — 1,1}) 
when p 7^ 1/2. The first point P_(oo) = 1 is stable while the second point P_(oo) = is 
unstable when p < 1/2, so the circuit computes the OR function of the variables belonging 
to the input set S 1 . For p > 1/2 the point P_(oo) = 1 is unstable and the point P_(oo) = 
is stable, so the circuit computes AND function. 

Secondly, we set p = 1/2 and allow for m(0) 7^ m(0), i.e. we have two copies of the same 
circuit but with different inputs in the equations f]37]) . f]38]) . For p = 1/2 the magnetization 
in the circuit is conserved from layer to layer (m(£) = m(0)) and the overlap equation (138]) 
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reduces to 



C(£ + 1) = \ + \C{1) - \m 2 (0) + ^m(0)m(0) - im 2 (0) + \c\£). (40) 

Using the above equation to compute the joint probability P e (— 1, 1) = (1 — m(0) + m(0) — 
C(£))/4 gives 



/*(-!, 1) = ^(-1,1) 



1 - i m (0) + iro(O) - J*(-M) 



(41) 



which is the equation of Lemma 3.2 in 



281 ] . In general, for £ — > oo, we have P°°(S, S 



1 + 5m(0) + Sm(0) + SSC (oo))/4, where C(oo) = 1 - |m(0) -m(0)|. The analysis |28j of 



equation ( l4ip reveals that AND/OR-based formulae compute constant ±1 functions when 
S 1 = {Si, . . . , S n , —Si, . . . , —S n } and linear threshold functions sgn[^™ =1 Sj — n + 2i], where 
i G {1, . . . , n} and S 1 = {Si, . . . , S n }. Finally, the convergence to the functions computed 
in this model is mainly exponential in £ except in one special case of p = 1/2 when it is 
logarithmic (<- 



VI. DISCUSSION 



We have presented the theoretical framework that allows one to study the typical proper- 
ties of noisy random Boolean formulae. A Boolean formula is a simple model of computation, 
which plays an important role in many areas of the theoretical computer science (TCS). The 
tree-like structure of Boolean formulae allows for the computation of exact noise thresholds 
by considering formulae with the worst possible topology. Another area of TCS where 
Boolean formulae played an important role is the generation of random Boolean functions. 
Here one usually uses a growth process to generate random formulae that induce a uniform 
probability distribution on the Boolean functions they compute. 

Here, for the first time, random formulae generated by the growth process are used to 
study the typical properties of noisy formulae. The method used here relies on the layered 
variant of the Savicky formula-growth process. The layered framework allows us a direct 
mapping to the physical Ising spin system, which can be seen as a dynamical system where 
the time-steps correspond to circuit layers. This analogy allows one to use the generating 
functional analysis (GFA) method of statistical physics. The GFA method has an excellent 
record in the area of disordered dynamical systems and is generally accepted to be exact. 
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Here, we use GFA to study the typical properties of noisy random Boolean formulae con- 
structed from single gates or distributions of gates. All exact noise thresholds, which were 
derived in TCS using rigorous methods, are recovered within our framework and identified 
with the corresponding macroscopic phase transitions. We attribute this to the exact corre- 
spondence of the mean-field equation (jl8p to the single-gate probability of error (assuming 
independence of inputs) used to derive these noise thresholds in TCS. However, many of the 
properties of noisy random Boolean formulae studied here are inaccessible via the traditional 
analytic methods of TCS and Information Theory. 

In the noiseless case (e = 0), we have identified the Boolean functions generated by the 
layered growth process, but only when the input-set magnetization m(0) = is not a fixed 
point of the dynamics ( fl8|) . For inputs with m(0) = our results are consistent with the 
results of the Savicky growth process that can generate a Boolean function of arbitrary 
complexity. Furthermore, we have established that the functions generated in the layered 
growth process are sensitive to the input variables, an indication of their complexity. In 
order to find out exactly which functions are generated for the input set with m(0) = 0, or 
in a more general setting with e > 0, one needs a more sophisticated version of the mean-field 



theory presented here 



29]. 



For e > we have studied the evolution of the output magnetization and computation- 
error from layer to layer and their dependence on the input-set magnetization and noise. We 
have identified a range of gate-noise parameter, and its dependence on the input-set magne- 
tization m(0), where by adding more layers to the circuit one can reduce the computation 
error. The speed of convergence to the equilibrium was studied both numerically (for e = 0) 
and analytically (for e > 0). For e = our numerical results are consistent with the rigorous 
bounds, but show that in a typical case a much tighter bound can be derived. 

The standard noisy computation model was expanded to include "production errors" 
(uncorrelated hard noise). We have found that the effect of hard noise on the critical 
behavior of noisy circuit is to effectively reduce its critical noise threshold. We expect that 
the critical noise threshold (if it exists) in any noisy circuit will be affected in this way. 
Also, in our work a standard flip-noise is compared with threshold noise. In particular, 
we have found that the MAJ-k gate is more resilient to threshold noise than to flip-noise. 
We expect that any gate which can be represented as the linear threshold function is more 
robust against the threshold noise, at least for the distribution of threshold noise considered 
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in this paper. 

We believe that much can be learned about the typical properties of noisy computation 
via this approach, which complements the rigorous bounds derived in the TCS literature 
and provides insight that may help in the development of new rigorous techniques. 
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Appendix A: Noise average 

In this appendix, we show for completeness how to derive the microscopic law (j2J) from the 
basic computation step in the von Neumann's model of noisy computation. The basic step in 
a noisy circuit is to compute the output of the (£, i)-th gate, given the input S^" 1 , • • • , S l ~ l , 
according to the stochastic rule 

sI = t£<4(s£\...,sI; 1 ) (Ai) 

where rjf is an independent random variable from the distribution P(rj) = e5 v --i+(l— e)5 m \. 
Equation OAip gives rise to the conditional probability 

P{S\\Sl~\ Si; 1 ) = (S shm e { s^,...,s^^ ■ (A2) 
Averaging out the noise variable produces the equation 

P&K 1 , • • • , Si; 1 ) = l[l+(l-2e)5fo{(5£- 1 , . . . , Ol > ( A3 ) 

which in turn leads to the equation (j2J) if one uses the transformation of noise variables 
1— 2e = tanh/3, where ft G [0, oo), and exploit the property — tanh(x) = tanh(— x). 
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Appendix B: Disorder average 



Here, we outline the calculation steps which lead to the saddle-point integral ( TTU1) . The 
starting point of this calculation is the generating functional 

' L,N 



{S e ,S e } 

L,N 

x exp 



e,i=i 



xe" 



{log2cosh[/3 J fff- 1 (5^ 1 )] +log2cosh[/3^- 1 (5' ')] 

l,i=l 



(Bl) 



where in the above we have defined the field terms H i (S 



it, A h,---,ik a i( S h 



St 1 ) and Ht\S 



£-1 



\N a£,i 1/ q- " Q 

^1 '■■■■> ° 3k 



e-i. 



■'31,— ,3k "Jl>— ,3k~"<- ^"31 ' " " " ' "3k ' " l y ~ I {—131,... ,j k "Ji,... ,3k' 

Enforcing the definitions of fields in the equation (1B1I) via the integral representations of 
unity 



N L-l 

nn 

t=l 1=0 



leads to 



dH { dx { ix i [Hf-Hf (s e )] 
2tt 



N L-l 

nn 

i=l 1=0 



N L-l 



dfffdxfofffdxf 



^ P{S\S°\S I )e-^Ms t Mim (B3) 

{S«,S«} 



t=l £=0 

JZi=l Y. L t Zv{PS t i + 1 Hl+pS l : +1 Hi-\og 2 co S h[/3Hf]-log 2 co 8 h[$Hf]+ixjHf+i&jHf} 



xe z 

L,JV V 

><n n 

e,i=lji,—,3k 



(B4) 
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For now we concentrate only on the last line of the equation (1B3[) that depends on the 
connectivity (J5J) and gate disorder (J7]). We average over the disorder in (IB4I) as follows 



L,JV AT 

£,i=lji,—,jk 

L,N 

= e n 



(B5) 



7T J | AT 



2tt 



f4"'' \t,!=l " " Hi — i** 

E D / A -Uf'* ■ {x e - 1 aHS e -\....S e - 1 )+x e - 1 a e (S e - 1 ,...,S e - 1 )+uj e }\ 

1 ^ 

_i_ ^ /g-'K <*(s h )+ w l}_i\ +o(JV-*) 



L,N 

n 

Li=l 



2tt 



4 e 1 ^ exp 



n,...,i k 



In the first line of the calculation we have used the integral representation of Kronecker 
delta function 5 n . m = ^L e ^(n-m) anc j j n ^ as i \[ ne fa e exponential form is valid for 
large N. By setting all the x\ and x\ variables to in the equation flB5p . we find that 
the normalization constant contributes the factor e NL to the generating functional (1B1[) 
when N is large. Using the result of disorder- average (1B5I) in the equation ( IBlh we obtain 
the disorder-averaged generating functional 

1 -e 1 ^ \ (B6) 



r ^*] = nn / 



(27T) 



2vr 



i=l £=Q 

x V p(5° ) 5 |5 / ) e - i ^^^ + ^ s ^ +7VL 

{S^,S«} 

Eili EtV {l3Sl- l H{+$Sf- 1 6i-loe 2 coBh[/BH<]-log 2 cosh[0Af]+taf 



xe z 



L 

x J^J exp 



N 



+0(A^- fc+1 

\ la 

In order to achieve factorization over sites in the equation we isolate the densities 

1 - 



'S;Sf 



i=l 



1 N 

1=1 



(B7) 
(B8) 
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via the respective integro-functional representations of unity 

, | d p^ dj p^ e W E S ,s^(S,S)[P < (5,5)-iE£ 1 '5 S;S |5 d;g |] = x 

J {df/df^je^^^^^ = 1 



(B9) 



which leads to 



r[t/>;<0] 

{dPdPdrJciri} exp 



(BIO) 



^=0 I 5,5 J 



x exp 



L-l 



^=o {Sj,Sj}3=i 



i{aa(Si,...,S fc }+sa(^i,...,§fcXw} 



X 



N 

n 

i=l 



L-l 



£=0 L 5f ; 5« 



(27T) 



2tt 



x ^S? +1 #f+/9Sf^iff-log2co^ 



3 0(LAf- fc+1 ) 



The site-dependent part of the above equation can be written in the form 
1 N f f 

Xi\ Si, Si 



exp 



n i=l 



(Bll) 



where we have defined the effective single-site measure 



M ni [Hi, Xi, Hi, Xi, u)i, Si, Si 



(B12) 



L-l 

xJTe 

1=0 



PS% +1 Hf +psf +1 Hf-log 2 cosh[/3tff ]-log 2 cosh[/3L>f ]+ixf i?f +ixf tff-iP^Sf ,Sf ,xf 



and we use the definition J Dcjj = Yli=i J- i^" e1 ^- Using the definition (IB lip in the 
disorder-averaged generating functional (IBlOp with all the generating fields {ipf, ipf} being 
set to and assuming that the law of large numbers for the random index- variables {n^} 
holds, i.e. lini7v->oo h Z)i=i — > P(n), we arrive at the result of equation ffTOl . 
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Appendix C: Simplification of the saddle-point problem 



In this appendix, we show how to solve the saddle-point equations (fT3|) - ([TE]) . First, we 
use the saddle-point equation ([To]) to eliminate the conjugate order parameter fr from the 
effective measure ffl2l) . giving 



M r . 



x exp 
xe -iP>^) 



e e ix e H +ix H +iui e+1 

e=0 2cosh(/3^)2cosh(/3^) 

k 



({5 i })+^a({5 J })W+ 1 } 



(CI) 



Second, using the above result we compute the Fourier transform 

F*[ y> z} = J {dHdxdHdxVu} M n [H, x; H, x; u>; 5, 5]e" i7{x 



(C2) 



5,5 



where 7 G {0, 1}. For 7 = we obtain 



L-l 



(C3) 



and for 7 = 1 we have 



L-l 



Ff [y, z}= J2 Wso;# +1 ; ^' +1 ]iV[^'; J] 5* +1 ]iV[^; (C4) 

{S e ,S e } 



where 



W[S i+1 ;S i+1 ] = £ ff[P^,S 

{5 J ,5 J }i= 1 

^+1.^+1]_ 



2 cosh(/3a({5 i }))2 cosh(/3a({S' j })) 



iV[5 £ ; S £} 



2cosh(/fy)2cosh(/3z) 



(C5) 

(C6) 
(C7) 



Next we notice that f dxdxdu£l e ' (x, x, u)e 1 i x v+ xz + u '} = ^ Using the fact that ( IC5D 



and ( 1C6P are both probability distributions the computation for £' = L — 1 gives -4r 



L-l 



Plugging in this result into the saddle-point equation (TT^|) gives us P (S, S) = ik implying 
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that JV[S ,L_1 ; S L_1 ] = e fe . The latter is used to show that for £' = L-2 gives 5tJ4 = 1 and 
so on until we conclude that P (S, S) = ik for all i. 
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